Font recognition using text localization

ABSTRACT

Font recognition and similarity determination techniques and systems are described. In a first example, localization techniques are described to train a model using machine learning (e.g., a convolutional neural network) using training images. The model is then used to localize text in a subsequently received image, and may do so automatically and without user intervention, e.g., without specifying any of the edges of a bounding box. In a second example, a deep neural network is directly learned as an embedding function of a model that is usable to determine font similarity. In a third example, techniques are described that leverage attributes described in metadata associated with fonts as part of font recognition and similarity determinations.

BACKGROUND

Creative professionals often utilize a variety of images as part ofcontent creation, such as to generate marketing materials, backgrounds,illustrate books, presentations, and so forth. For instance, creativeprofessionals may create images themselves which are then included inthe content, such as for part of a presentation, and may also obtainimages from outside sources, such as from a content sharing service.Accordingly, even a single item of content may include a variety ofimages obtained from a variety of different sources.

In some instances, these images include text, such as text on a roadsign, a person's shirt, a logo, and so forth. Text, and the fonts usedto render the text in the image, are one of the top elements of design.Accordingly, recognition of a font used to render text within an imageand also to find similar fonts (e.g., to promote a similar look and feelto an item of content) is an important factor in creation of contentthat is visually pleasing to users. Conventional techniques to do so,however, typically rely on manual user interaction on the part of thecreative professional, which may introduce errors due to reliance on themanual dexterity of the user that performs this interaction. Althoughautomated techniques have been developed, these are often also prone toerror, resource intensive, and inefficient and thus limited to deviceshaving sufficient processing resources to perform these conventionaltechniques.

SUMMARY

Font recognition and similarity determination techniques and systems aredescribed. In a first example, localization techniques are described totrain a model using machine learning (e.g., a convolutional neuralnetwork) using training images. The model is then used to localize textin a subsequently received image, and may do so automatically andwithout user intervention, e.g., without specifying any of the edges ofa bounding box. In a second example, a deep neural network is directlylearned as an embedding function of a model that is usable to determinefont similarity. In a third example, techniques are described thatleverage attributes described in metadata associated with fonts as partof font recognition and similarity determinations.

This Summary introduces a selection of concepts in a simplified formthat are further described below in the Detailed Description. As such,this Summary is not intended to identify essential features of theclaimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different instances in thedescription and the figures may indicate similar or identical items.Entities represented in the figures may be indicative of one or moreentities and thus reference may be made interchangeably to single orplural forms of the entities in the discussion.

FIG. 1 is an illustration of an environment in an example implementationthat is operable to employ font recognition and similarity determinationtechniques described herein.

FIG. 2 depicts a system in an example implementation showing a textlocalization system of FIG. 1 in greater detail.

FIG. 3 is a flow diagram depicting a procedure in an exampleimplementation in which text is localized for use in font recognitionand similarity determinations.

FIG. 4 depicts an example implementation showing rendered trainingexamples with different perturbations.

FIG. 5 depicts upper and lower boundaries used as a ground truth formachine learning.

FIG. 6 depicts examples of overlapping crops used for text localization.

FIG. 7 depicts an example implementation of localization results withline fitting.

FIG. 8 depicts a system in an example implementation showing a textsimilarity system of FIG. 1 in greater detail.

FIG. 9 is a flow diagram depicting a procedure in an exampleimplementation that is used to train and use a model to determine fontsimilarity.

FIG. 10 depicts example results of the techniques described hereincontrasted with conventional results.

FIG. 11 depicts a system in an example implementation showing a fontattribute system of FIG. 1 in greater detail.

FIG. 12 is a flow diagram depicting a procedure in an exampleimplementation that is used to train and use a model to determine fontsand font similarity by using attributes associated with the font.

FIG. 13 depicts an example implementation showing features learned forma weight attribute network contrasted with a font recognition feature.

FIG. 14 depicts an example implementation showing italic attributefeatures learned from a regular-italic attribute network functioncontrasted with font recognition features.

FIG. 15 illustrates an example system including various components of anexample device that can be implemented as any type of computing deviceas described and/or utilize with reference to FIGS. 1-14 to implementembodiments of the techniques described herein.

DETAILED DESCRIPTION

Overview

Fonts used to render text in images and other content are one of the topelements in content design. As fonts are used to present text incontent, a user's interaction with the font and text is typicallygreater than that with other objects in the content as the usertypically closely reads the text and then observes the other objects. Assuch, choice of fonts as well as visual consistency between fonts withincontent is one of the most important factors in how the content isperceived by a perspective audience.

Conventional digital medium environments used to create content,however, support a limited ability to recognize fonts used to rendertext within an image as well as to locate similar fonts, such as topromote consistency in appearance in text rendered using the fonts inthe content. For example, conventional techniques may rely on manualselection of a portion of an image (e.g., by manually drawing a boundingbox that surrounds the portion) that is to be processed to recognize afont used to render text in the bounding box. As such, this conventionaltechnique is limited by accuracy of the manual selection in order todrawn the bounding box and corresponding dexterity of a user doing so.

Accordingly, text localization techniques are described in which adigital medium environment is configured to localize text in an imagefor an arbitrary font. These techniques also have increased accuracy(e.g., an improvement of approximately double as further describedbelow), have increased efficiency such that these techniques take lessthan 0.1 second to process an image for a single word on a consumergraphics processing unit, and are not limited to a small set of simplefonts as in conventional automated techniques.

In order to so do, text localization techniques described herein train amodel using machine learning (e.g., a convolutional neural network)using training images. The model is then used to localize text in asubsequently received image, and may do so automatically and withoutuser intervention, e.g., without specifying any of the edges of thebounding box. In this way, manual specification of the bounding box inconventional techniques is avoided along with the inaccuracies involvedin doing so. Further, these techniques are able to address arbitraryfonts and thus are not limited to a small set of particular fonts inconventional automated techniques, additional discussion of which isdescribed in relation to FIGS. 2-7 in the following.

Also techniques employed by conventional digital medium environments donot support a mechanism to locate similar fonts. Rather, conventionaltechniques rely solely on classification and not similarity, and thus isfocused on discriminating between different fonts rather than retrievingsimilar fonts. Thus, these conventional techniques are prone toinaccuracies if used for purposes other than classification.Accordingly, techniques are described herein in which font similarity isused to find visually similar fonts for a given font. For instance, fontsimilarity may be used to determine which fonts are similar to a fontused to render text in an image, which may be used to navigate throughhundreds or even thousands of fonts to find a similar font of interest.In this way, a user may navigate through a vast collection of fonts tolocate a font of interest based at least in part on similarity of fontsto a font used to create content in an efficient, intuitive, andaccurate manner that promotes visually pleasing content.

In the following, a deep neural network is directly learned as anembedding function of a model that is usable to determine fontsimilarity. Techniques are also described in which data is sampled withincreased efficiency to expedite the learning process. Furtherdiscussion of these and other examples are also contemplated, furtherdescription of which is included in relation to FIGS. 8-10 below.

Further, techniques employed by conventional digital medium environmentsare limited and ignore potentially useful information in an attempt toperform font recognition and/or similarity. As described above, fontrecognition involves the challenge of recognizing the font of text froman image, whereas font similarity involves the challenge of findingvisually similar fonts for a given font. Both techniques are extremelyuseful for creative professionals in font selection.

Conventional techniques rely solely on an appearance of the fontsthemselves to determine similarity and thus ignore other potentiallyuseful information in making this determination. Accordingly, techniquesare described herein that leverage attributes (e.g., described inmetadata) associated with fonts as part of font recognition andsimilarity determinations, examples of which are described in furtherdetail in relation to FIGS. 8-14.

In the following discussion, an example environment is first describedthat may employ the techniques described herein. Example procedures arethen described which may be performed in the example environment as wellas other environments. Consequently, performance of the exampleprocedures is not limited to the example environment and the exampleenvironment is not limited to performance of the example procedures.

Example Environment

FIG. 1 is an illustration of an environment 100 in an exampleimplementation that is operable to perform text localization, image fontrecognition, and image font similarity techniques described herein. Theillustrated environment 100 includes a computing device 102, which maybe configured in a variety of ways.

The computing device 102, for instance, may be configured as a desktopcomputer, a laptop computer, a mobile device (e.g., assuming a handheldconfiguration such as a tablet or mobile phone as illustrated), and soforth. Thus, the computing device 102 may range from full resourcedevices with substantial memory and processor resources (e.g., personalcomputers, game consoles) to a low-resource device with limited memoryand/or processing resources (e.g., mobile devices). Additionally,although a single computing device 102 is shown, the computing device102 may be representative of a plurality of different devices, such asmultiple servers utilized by a business to perform operations “over thecloud” via a network 104 as further described in relation to FIG. 15.

The computing device 102 is illustrated as including a variety ofhardware components, examples of which include a processing system 106,an example of a computer-readable storage medium illustrated as memory108, a display device 110, and so on. The processing system 106 isrepresentative of functionality to perform operations through executionof instructions stored in the memory 108. Although illustratedseparately, functionality of these components may be further divided(e.g., over the network 104), combined (e.g., on an application specificintegrated circuit), and so forth.

The processing system 104 is illustrated as executing an image editingmodule 112 which are storable in the memory 106 and as such isimplemented at least partially in hardware. The image editing module 112is executable by the processing system 106 to cause performance of oneor more operations. Other implementations are also contemplated, such asimplementation as dedicated hardware components, e.g., applicationspecific integrated circuit, fixed-logic circuitry, and so forth.

The image editing module 112 is representative of functionality of thecomputing device 102 to create (e.g., originate and/or modify) andmanage images 114 through interaction with a user interface 116displayed by the display device 110. For example, a user may use akeyboard, cursor control device, gesture detected by touchscreenfunctionality of the display device 110, verbal utterance, and so on tointeract with the image 114, an example of which is rendered image 118in the user interface 116 on the display device 110. The image 114 caninclude a variety of different objects, such as text, shapes or othervisual objects, spreadsheets, as a document, a multimedia content, slidepresentation, and so on.

An example of functionality to create and edit images is illustrated asa font recognition and similarity system 120. This system 120 isrepresentative of functionality to perform text localization, findsimilar fonts, and employ font attributes for font recognition andsimilarity, examples of which are represented by the text localizationsystem 122, font similarity system 124, and font attribute system 126,respectively.

The text localization system 122 is representative of functionality tolocalize text 128 within an image 114, such as to locate the text “keepit simple and smart” for the rendered image 118 in the user interface116. The localized text, for instance, may be included in a bounding boxthat is automatically defined by the text localization system 122without user intervention. The localized text is then used to recognizewhich of a plurality of fonts 130 are used to render the text, findsimilar fonts 130, and so on. Examples of fonts 130 are illustrated instorage 132 of the computing device 102 but may also be maintained overthe network 104 as previously described. As the bounding box defineswhich pixels are to be processed to recognize fonts and/or determinefont similarity, accuracy of the bounding box is an important factor inthe accuracy of this processing. The techniques described herein similarhave improved accuracy and processing efficiency as not being limited bymanual dexterity of a user that draws the box or other conventionaltechniques that are limited to specific fonts, further discussion ofwhich is described in relation to FIGS. 2-7 of the next section.

Another example of functionality of the font recognition and similaritysystem 120 is represented by the font similarity system 124, which isusable to determine which fonts 130 are similar to fonts used to rendertext 128 in an image 114. For instance, font similarity may be used tonavigate through hundreds or even thousands of fonts to find a similarfont of interest. To do so, a deep neural network is directly learned asan embedding function of a model that is usable to determine fontsimilarity, which is then used to locate and view similar fonts asdesired by a user. Techniques are also described in which data issampled with increased efficiency to expedite the learning process.Further discussion of these and other examples are also contemplated,further description of which is included in relation to FIGS. 8-10below.

A further example of functionality of the font recognition andsimilarity system 120 is represented by the font attribute system 126,which is usable to employ attributes from metadata as part of fontrecognition and/or a determination of font similarity. Attributes aretypically defined by a designer of the font and may include relativeattributes that are usable to define a relationship of fonts within afont family to each other (e.g., weight, regular/italics pairs) andcategorical attributes that describe characteristics generally, e.g.,Serif versus Sans-Serif. These attributes are learnable as part of amachine learning process to improve accuracy and efficiency of fontrecognition and similarity determinations, further discussion of whichis included in the following in relation to FIGS. 8-14.

Having described a digital medium environment that is operable to employthe techniques described herein, discussion is now made in the followingsections further providing examples of functionality represented by thetext localization system 122, font similarity system 124, and fontattribute system 126, respectively.

Improved Font Recognition Using Text Localization

FIGS. 2-7 depicts examples of the text localization system 122 of FIG. 2in greater detail. The text localization system 122 addresses thechallenge of finding and defining a boundary surrounding text that isincluded in an image. In one example, localization involves computationof a bounding box of the text in an input image, which describes whatportion of the image includes the rendered font. The bounding box thenserves as a basis for font recognition and similarity determinations asdefining which pixels in the image include the rendered text that is toserve as a basis for processing performed to support thesedeterminations.

This has particular importance in content creation as use of particularfonts is one of the top elements in design, for as previously describeduser interaction with text and fonts used to render text is typicallyhigher than with other objects in an image. Thus, determination of whichfonts are used to render text in an image and determination of whichfonts are similar to the determined fonts are important factors incontent creation. Accordingly, accuracy of text localization used tosupport these techniques is an equally important factor.

For example, a user may select an image having text for inclusion aspart of content, e.g., a marketing campaign. In order to keep aconsistent look and feel to the content, font recognition may beperformed to determine which font is used to render the text and fontsimilarity may be performed to locate fonts that are similar to thisfont. In this way, a creative professional may include additional textas part of creating the content that has a similar look and feel bycausing the text to be rendered using the same or similar fonts.

There are two kinds of related conventional algorithms: text detectionand optical character recognition. Both of these conventional algorithmsare used to detect presence of text in an image and are typicallylimited to a small set of predefined fonts. As such, these conventionalalgorithms often fail for complicated fonts such as script fonts anddecorative fonts and thus have limited applicability in real worldapplications.

In the following, techniques and systems are described to automaticallylocalize a text region in an image to improve accuracy in definingboundaries of a bounding box. These techniques employ machine learning(e.g., a deep convolutional neural network) and thus exhibit improvedresource consumption (e.g., both in amount of resources used and timeused by those resources) and are usable for arbitrary fonts. Thus, thesetechniques may be performed to localize text without knowledge of whichparticular fonts are being used to render the text and thus expandsapplicability to a wider range of fonts. For example, script anddecorative fonts may be processed using the techniques described hereinwhich is not possible using conventional text detection or opticalcharacter recognition algorithms as described above. In one or moreimplementations, these techniques are usable to significantly improvefont recognition accuracy, from a conventional accuracy of 40.2% and59.2% for top-1 and top-5 tiers, respectively, to accuracies of 76.3%and 93.3% for top-1 and top-5 tiers. Thus, a 90% improvement may beobserved in terms of the top-1 accuracy with improved resourceconsumption using the techniques described below.

FIG. 2 depicts a system 200 in an example implementation showing thetext localization system 122 of FIG. 1 in greater detail and FIG. 3depicts a procedure 300 in an example implementation that is used totrain and use a model to perform text localization. In the following,reference is made interchangeably to both FIGS. 2 and 3.

The following discussion describes techniques that may be implementedutilizing the previously described systems and devices. Aspects of theprocedure may be implemented in hardware, firmware, software, or acombination thereof. The procedure is shown as a set of blocks thatspecify operations performed by one or more devices and are notnecessarily limited to the orders shown for performing the operations bythe respective blocks.

The text localization system 122 is illustrated as including a textlocalization training system 202 and a text localization module 204. Thetext localization training system 202 is representative of functionalitythat is usable to generate a text localization model 206, e.g., viamachine learning. The text localization model 206 is then used by thetext localization module 204 to localize text 128 in an image 114, e.g.,to find a bounding box of the text for font recognition or similaritydeterminations. As the bounding box defines which pixels are to beprocessed to perform this determination, accuracy of the bounding box incapturing rendered text is a primary component in the accuracy of thefont recognition determination and font similarity determination.

A machine learning approach is taken by the text localization system 122to train the text localization model 206 by the text localizationtraining system 202 and to perform text localization using the model bythe text localization module 204. By doing so, the text localizationsystem 122 is able to address the tens of thousands of fonts andcorresponding text appearance changes for rendered text 128 in an image114, which is not possible using conventional hand-designed algorithmsthat are limited to particular fonts. Thus, even though a relativelymoderate number of general font types may be used to generate the textlocalization module 206 in this example, the text localization model 206may be employed to perform text localization even for other fonts notwere not used, particularly, to train the model. In one or moreimplementations, however, the text localization model 206 may befine-tuned on a particular set of fonts and application settings. Forinstance, the text localization training system 202 can train the textlocalization model 206 for images taken on a cellphone by synthesizingimages according to the cellphone imaging process, e.g., by addingparticular JPEG compression, and so forth.

In the text localization system 122 there is a training phase in whichthe text localization model 206 is generated and a testing phase inwhich the model is employed for an input image. In this example, thetext localization training system 202 is utilized offline and the testphase involving use of the image 114 is performed in real time, althoughother examples are also contemplated. State-of-the-art convolutionalneural networks are used as the architecture. Batch based stochasticgradient descent is used as a training algorithm in the following,although other examples are also contemplated. This may be used toperform text localization to provide precise bounding boxes, and may beused to train the model to match settings of the font recognitiontechniques described herein to increase accuracy as further describedbelow.

To begin, a training set generation module 208 is utilized to generate atraining image and font collection 210 that includes training imagesthat are to serve as a basis for training the text localization model206. The training image and font collection 210, for example, may begenerated as synthetic images using fonts 130. For instance, thetraining set generation module 208 renders text in images using aselection of fonts 130 and may add perturbations (e.g., rotations, skew,and so on) as further described below to generate the training image andfont collection 210.

For example, in order to make the training set more diversified and morerobust to noises, random perturbations may be added by the training setgeneration module 208 during and/or after rendering of text. Examples ofperturbations include kerning offset, downscaling, background/foregroundintensity, text color flipping, shading, rotation, squeezing, croppingand noise. An example implementation of rendering parameters is listedin the following:

Perturbation parameter Value Comments pert_ns 10 Noise variance pert_sc1.5, 2.0 Downscaling factor range pert_sh 0.2, 0.5 Shading gradientrange kerning −5, 350 Kerning offset range pert_bg 130, 200 Backgroundcolor range pert_fl 1 Black/white flipping pert_fg 50, 120 Foregroundcolor range pert_rt 4 Maximum rotation angle squeeze 1.5, 2.5, 3.5Squeezing factors pert_mg −70, 0 Margin range (outside text) font_size200 Font size

FIG. 4 depicts an example implementation 400 showing rendered trainingexamples with different perturbations selected from the table above.Other examples are also contemplated, such as to form the training imageand font collection 210 from “real” images that are not syntheticallygenerated.

Regardless of how generated, a machine learning module 212 obtains thetraining image and font collection 210. The collection includes aplurality of training images having text rendered using a correspondingfont (block 302). A model is trained by the machine learning module 212to predict a bounding box for text in an image. The model is trainedusing machine learning as applied to the plurality of training imageshaving text rendered using the corresponding font (block 304).

Normalized y-coordinates of upper boundary “y_(u)” 502 and baseline“y_(b)” 504 are used as the ground truth for machine learning as shownin an example implementation 500 of FIG. 5. For example, the upperboundary 502 is the highest horizontal line of the text area and thelower boundary 504 is the baseline of text. The upper and lowerboundaries “y_(u)” and “y_(b)” 502, 504 are normalized by the originalimage height. The x-coordinates of the bounding box may be representedsimilarly and are omitted in the following discussion for the sake ofsimplicity of the discussion.

The machine learning module 212 may employ a variety of differenttechniques to train the text localization model 206, an example of whichis a convolutional neural network. To train this network, the initiallearning rate is set to 0.01 for the machine learning module 212 in thisexample and is reduced to 0.001 in the middle of training. Batch size isset to 128, momentum is set to 0.9 and weight decay is set to 0.0005 forall the layers. Dropout ratio after fc7 and fc8 is set to 0.5. Thenetwork structure is listed as following:

Name type Kernel size/stride Output size Input input 111 × 111 × 1 conv1convolution 11 × 11/2 51 × 51 × 64 pool1 pooling 3 × 3/2 25 × 25 × 64conv2 convolution 5 × 5/1 21 × 21 × 128 pool2 pooling 3 × 3/2 10 × 10 ×128 conv3 convolution 3 × 3/1 10 × 10 × 256 conv4 convolution 3 × 3/1 10× 10 × 256 conv5 convolution 3 × 3/1 10 × 10 × 256 pool5 pooling 5 × 5/52 × 2 × 256 fc7 fully connected 1 × 1 × 1024 fc8 fully connected 1 × 1 ×2

A squared l2 loss is used as the loss function, an example of which isexpressed as follows:loss=∥y−f(x)∥²,in which “y” is normalized ground truth, and “f(x)” is the output of“fc8.” The network is learned by a stochastic gradient decent technique.

The machine learning module 212 may employ a variety of differenttechniques to train the convolutional neural network. In a first suchexample, the machine learning module 212 starts from a randominitialization of the neural network. In a second such example, themachine learning module 212 starts from a classification network whichis trained for a classification task. An advantage may be observed bystarting from the classification network and this technique tends toconverge faster as there are fewer parameters to learn. Additionally, ifthe convolutional layers are fixed then the network is smaller as theparameters are shared with the classification network and thus exhibitsimproved efficiency. Thus, at this point the text localization trainingsystem 202 has generated a text localization model 206 that is usable tolocalize text 128 in a received image 114 as further described below.

The text localization module 204 of the text localization system 122then receives the text localization module 206 for testing, i.e., tolocalize text 128 in a received image 114. Thus, the text localizationmodule 204 first obtains the model 206 that is trained using machinelearning as applied to a plurality of training images having textrendered using a corresponding font (block 306). A bounding box 214 fortext 128 in an image 114 is predicted using the obtained model 206(block 308) and an indication is generated of the predicted bounding box214. The indication is usable to specify a region of the image thatincludes the text having a font to be recognized (block 310).

For example, a horizontal squeeze (2.5×) is first applied to the image114 at test time, an amount of which matching a training setting of thetext localization training system 202 to generate the text localizationmodel 206. This is used to improve processing efficiency and accuracy.

Overlapping crops 602, 604, 606 are then formed from the squeezed imageas shown in an example implementation 600 of FIG. 6. The cropped imageshave predefined sizes (e.g., 111×111 pixels) and are denoted as “x₁, x₂,. . . , x_(M)” in the following. Each cropped image 602, 604, 606 isthen fed independently into the trained convolutional network of thetext localization model 206. The text localization model 206, throughprocessing using the text localization module 204, is thus used toobtain bounding box prediction for each cropped image 602, 604, 606,which are represented by values of “f(x₁), f(x₂), . . . , f(x_(M))” inthe following.

The text localization module 204 then generates a resulting bounding box214, which may be calculated in a variety of ways. In a first example,an average is calculated for both top and bottom lines for the boundingbox predictions obtained for the cropped images 620, 604, 606, e.g.,upper boundary and baseline as described above. In another example, amedian is calculated for both top and bottom lines for the bounding boxpredictions obtained for the cropped images 620, 604, 606. A linefitting algorithm may also be used to fit lines to both top and bottomlines separately or jointly from the cropped images, which may also beused to determine a rotation of text 128 in the image 114.

FIG. 7 depicts an example implementation of localization results 702,704, 706, 708, 710, 712, 714, 716 with line fitting. The lines areindications of the upper and lower boundaries as indicated by thebounding box 214. Dots illustrated along these lines represent predicted“y” locations of each of the crops. The last two localization results714, 716 are obtained by processing real text images with irregularlayouts, which are accurately addressed by the text localization system122.

Through use of machine learning as described above, the textlocalization module 206 is usable to localize the text 128 in an image114 for arbitrary font, which as previously described is not possibleusing conventional techniques that are limited to a small set of simplepredefined fonts, and thus often fail for script fonts, decorativefonts, and so forth. These techniques are also resource efficient andfast, and are able to process a single word of text in an image 114 inless than 0.1 second. Furthermore, these techniques also improve therecognition accuracy significantly, going from (40.2%, 59.2%) top-1 andtop-5 tier accuracy to (76.3%, 93.3%), which is 90% improvement in termsof the top-1 accuracy. Having now described an example of textlocalization for font recognition and similarity, an example involvingvisual font similarity is described in the following section.

Determination of Font Similarity

FIG. 8 depicts a system 800 in an example implementation showing thetext similarity system 124 of FIG. 1 in greater detail and FIG. 9depicts a procedure 900 in an example implementation that is used totrain and use a model to determine font similarity. In the following,reference is made interchangeably to both FIGS. 8 and 9.

The following discussion describes techniques that may be implementedutilizing the previously described systems and devices. Aspects of theprocedure may be implemented in hardware, firmware, software, or acombination thereof. The procedure is shown as a set of blocks thatspecify operations performed by one or more devices and are notnecessarily limited to the orders shown for performing the operations bythe respective blocks.

Font similarity involves the challenge of finding visually similar fontsfor a given font. As previously described, as font choice is one of themost important design considerations in creating content, location ofsimilar fonts is equally important in content creation. For instance,font similarity may be used to determine which fonts are similar to afont used to render text in an image, which may be used to navigatethrough hundreds or even thousands of fonts to find a similar font ofinterest. In this way, a user may navigate through a vast collection offonts to locate a font of interest based at least in part on similarityof fonts to a font used to create content in an efficient, intuitive,and accurate manner.

Conventional techniques used to process fonts are based on training aconvolutional neural network for font classification. In other words,these conventional techniques are not directly aimed at font similaritybut rather are targeted at discriminating between different fonts ratherthan retrieving similar fonts and accordingly is inaccurate for thispurpose. Therefore, the performance of these conventional techniques israther limited, as opposed to the techniques described herein that aredirectly aimed at learning features for font similarity.

Techniques and systems are described to determine font similarity bycomparing features from a learned network for different fonts. Ratherthan learn a network for classification and using a feature of thenetwork for classification as performed in conventional techniques, adeep neural network is directly learned as an embedding function of amodel that is usable to determine font similarity. Techniques are alsodescribed in which data is sampled with increased efficiency to expeditethe learning process.

First, training images are selected using font metadata associated withrespective fonts used to render text included in respective trainingimages (block 902). The font similarity system 124 employs a datasampling scheme which is computationally efficient and customized toaddress the font similarity problem. The space of the training data iscomposed of triplet combinations of images, which is too large toenumerate as a whole. Accordingly, font metadata is employed to design atriplet sampling distribution to increase a likelihood that relevanttriplets from the training set are used for feature computation andmodel training.

As an overview, the font similarity system 124 is configured to learn anembedding function for font similarity using machine learning and theentire system is learned end-to-end. To do so, the font similaritysystem 124 takes as inputs training images 802 having text renderedusing different fonts 130. The embedding function of the model is thenautomatically learned without user intervention when the neural networkis optimized using back-propagation. Training images 802 may take avariety of forms, such as synthetic or real images as described in theprevious section.

The neural network of the font similarity system 124 includes threeidentical columns in this example, which are represented by an anchorimage machine learning module 804, a positive image machine learningmodule 806, and a negative image machine learning module 808. The anchorimage machine learning module 804, positive image machine learningmodule 806, and negative image machine learning module 808 areconfigured to process an anchor image 610, positive image 812, andnegative image 814 (blocks 904, 906, 908) having characteristics thatare described in greater detail below. Each column may have a structureas shown in the table below.

Kernel size/ Name Type Stride Output size Input input 111 × 111 × 1conv1 convolution 11 × 11/2 51 × 51 × 64 pool1 pooling 3 × 3/2 25 × 25 ×64 conv2 convolution 5 × 5/1 21 × 21 × 128 pool2 pooling 3 × 3/2 10 × 10× 128 conv3 convolution 3 × 3/1 10 × 10 × 256 conv4 convolution 3 × 3/110 × 10 × 256 conv5 convolution 3 × 3/1 10 × 10 × 256 pool5 pooling 5 ×5/5 2 × 2 × 256 fc7 fully connected 1024 fc8 fully connected  256 NormL2 normalization  256

The three columns are constrained to be identical in both structure andparameter, i.e., there is only a single set of parameters to learn. Theloss function learned to form the model may be expressed as follows:

$\sum\limits_{k = 0}^{n}\;\left\lbrack {{{{f\left( x_{A} \right)} - {f\left( x_{P} \right)}}} - {{{f\left( x_{A} \right)} - {f\left( x_{N} \right)}}} + \alpha} \right\rbrack_{+}$where “x_(A),” “x_(P)” and “x_(N)” are the anchor image, positive image,and negative image respectively, and “α” is a parameter used to controla permissible margin of differences between positive image and negativeimage. The anchor image 812, as the name implies, is a training imagethat is to be used as a basis for comparison as part of the machinelearning with the positive and negative images 821, 814. The positiveimage “x_(P)” 812 has a font type asx_(A) that matches a font type ofthe anchor image 812, but is rendered with different text or randomperturbation as described in the previous section. The negative image“x_(N)” has a different font type than that of the anchor image 810.

The font similarity system 124 is configured obtain fast convergence(training speed) in the training of the model by selecting triplets(i.e., training images 802 that include the anchor, positive, andnegative images 810, 812, 814) that violate a triplet constraint. Toachieve this, the font similarity system 124 employs metadata 816associated with the fonts 130. The metadata 816 is typically assigned toeach font 130 are part of designing the fonts 130. The metadata 816 maydescribe a variety of characteristics of the fonts 130, such as a family(i.e., type) to which the font 130 belongs, line weight, whether regularor italic, recommended use, calligraphy style, and so forth. Themetadata 816 is categorical and can be encoded as a binary vector “m”where a zero/one entry indicates the presence/absence of a certain fontproperty.

When sampling a triplet of images, an anchor image “x_(A)” 810 is firstuniformly sampled from each of the fonts 130 and corresponding fontmetadata 816 is denoted in the following as “m_(A).” The positive image“x_(P)” 812 is then uniformly sampled by the font similarity system 124from the same font type as a font type used for the anchor image “x_(A)”810. This step is efficient because the number of images belonging tothe same font type is relatively small. Lastly, the font similaritysystem 124 samples the negative image 814 font “F_(N)” with metadata“m_(N)” from each of the font types that is different from the font typeof the anchor image “x_(A)” 810 with the following probabilitydistribution:

${p\left( F_{N} \right)} = {\frac{1}{Z}e^{{- \beta} \cdot {d{({m_{A},m_{N}})}}}}$where “d(m_(A), m_(N))” is a Hamming distance between the metadatavectors, and “β” is a positive coefficient. The value “Z” is anormalization factor defined as follows:

$Z = {\sum\limits_{F_{N} \neq F_{A}}\; e^{{- \beta} \cdot {d{({m_{A},m_{N}})}}}}$

The final negative image “x_(N)” 814 is randomly drawn from the fonttype “F_(N).” In one or more implementations, the distance matrix“{d(m₁, m₂)}” is pre-calculated and thus negative images are sampled inan efficient manner.

The training of the font similarity model 818 is then controlled by thefont similarity system 124 as an embedding function for font similarityas part of machine learning using the anchor image, the positive image,and the negative image (block 908). To train the model 818 using a deepconvolutional neural network, for instance, the initial learning rate isset to 0.01 and is reduced to 0.001 in the middle of training. Batchsize is set to 128. Momentum is set to 0.9 and weight decay is set to0.0005 for each of the layers in the deep convolutional neural network.Dropout ratio after layers “fc7” and “fc8” of the deep convolutionalneural network is set to 0.5. The margin “a” is set to 0.1 and “a” isset to be the inverse of the length of metadata 816 binary vector.

The font similarity model 818 is then obtained by the font similaritymodule 820 that was trained using machine learning as applied to theplurality of images (block 910). For example, the font similarity model818 may be trained offline by the font similarity system 124 and thenused in real time to process an image 114. A determination is then madeby the font similarity module 820 using the obtained model 818 as tosimilarity of a font used for text 128 in an image 114 with respect to aplurality of fonts 130 (block 912), which may include text localizationas described in the previous section.

Output of a result 822 of the determined similarity in a user interfaceis controlled (block 914) by the font similarity system 124. The fontsimilarity system 124, for instance, may output a list of fonts that aresimilar, apply the fonts to selected text, and so forth in a userinterface output by the computing device 102. An example of results 1002in shown in an example implementation 1000 depicted in FIG. 10 ascontrasted with a conventional result 1004. The figure shows an exampleof retrieving similar fonts to “AauxNext-SemiBoldItalic” using both afont recognition feature and the described triplet network feature. Inthe top 5 retrieval results, the techniques described herein find morefonts with “SemiboldItalic” or “MediumItalic” styles, which are visuallymore similar to the query font. A variety of other examples are alsocontemplated.

Font Attributes for Font Recognition and Similarity

FIG. 11 depicts a system 1100 in an example implementation showing thefont attribute system 126 of FIG. 1 in greater detail. FIG. 12 depicts aprocedure 1200 in an example implementation that is used to train anduse a model to recognize fonts and determine font similarity by usingattributes taken from metadata associated with the fonts. In thefollowing, reference is made interchangeably to both FIGS. 11 and 12 andcontinues on to a discussion of examples shown in FIGS. 13 and 14.

The following discussion describes techniques that may be implementedutilizing the previously described systems and devices. Aspects of theprocedure may be implemented in hardware, firmware, software, or acombination thereof. The procedure is shown as a set of blocks thatspecify operations performed by one or more devices and are notnecessarily limited to the orders shown for performing the operations bythe respective blocks.

The font attribute system 126 is representative of functionality toemploy metadata 816 that describes font attributes to improve visualfont recognition and font similarity. As previously described, fontrecognition involves the challenge of recognition of the font used torender text in an image whereas font similarity involves the challengeof finding visually similar fonts for a given font. By using visual fontsimilarity, a designer can easily browse hundreds or even thousands offonts and find a desired font.

Techniques are described in the previous section in which a deepconvolutional neural network is trained to recognize and find similarfonts. In the current example, the font attribute system 126 employsmetadata 816 in order to improve accuracy and efficiency of thesetechniques. For instance, within a font family, different fonts 130 havedifferent weights; fonts may have a notation of relative and italic; maycome as pairs; may have classification information such as Serif, SanSerif; and so forth. Accordingly, these attributes may also be leveragedto recognize a font having the described attributes as well as locatesimilar fonts having similar attributes as defined by associatedmetadata 816.

In the illustrated example, two types of attributes are defined by themetadata 816, relative attributes 1102 and categorical attributes 1104.Relative attributes 1102 describe relative characteristics of the fonts(e.g., line weight) and are usable to compare fonts within a font familyand with the processing described herein are also usable to compare withfonts outside the family. For instance, font families typically orderfonts by weights, by matching relative and italic pairs, and otherrelative attributes 1102. Accordingly, these relative attributes 1102are usable to describe a relationship of one font within a family toother fonts within the family. Further, these relative attributes 1102may also be quantified/normalized as further described below forcomparison with fonts in other families.

On the other hand, categorical attributes 1104 describe generalcategories that are usable to describe the fonts, e.g., classificationlabels such as “Serif.” However, categorical attributes 1104 can beambiguous because the attribute is defined by a designer of the font andthus may lack standardization with other designers and may not beavailable in some instances. Accordingly, techniques are described inthe following that employ metadata 816 to exploit relative attributes1102 to improve both font recognition and font similarity. However, itshould also be apparent that categorical attributes 1104 may also beused in some instances to improve font recognition and similaritydeterminations as this data, when available, may still be useful as partof these determinations.

In one example, a weight prediction function is first described in thefollowing that is employed by the font attribute system 126. The weightprediction function is usable to learn a relative weight attribute anduse this attribute for font recognition and similarity determinations.This function is first learned to compare fonts in the same font familybecause the weight attribute across different font families is typicallynot directly comparable as this value is defined by a designer of thefont and thus typically lacks standardization with other designers.After learning, however, the function may then be applied to fonts fromdifferent families as further described below.

In another example that is described in greater detail below, aregular-italic classifier is employed by the font attribute system 126that is usable to determine whether a pair of fonts are a matching pairof relative and italic fonts 130. This function is also first learned bythe font attribute system 126 to compare fonts in the same family.Again, after learning, the classifier can then be applied to fonts fromdifferent families by the font attribute system 126. In yet anotherexample that is further described in the following, the font attributesystem 126 employs a unified multi-task Siamese neural network structurethat is able to incorporate categorical and relative attributes for bothfont recognition and similarity.

In this section, metadata 816 that describes font attributes is used toimprove font recognition and similarity. To begin, attributes areselected that are to be used for this task because attributes are notequally useful in terms of improving font recognition and similarity.For instance, if an attribute solely applies to a small number of fonts,although the attribute may be useful in terms of improving recognitionand similarity for those particular fonts, the attribute may notgeneralize well to the fonts to which the attributes do not apply.Therefore, criteria that may be used in attribute selection are based onwhether the attribute applies to a relatively large number of fonts.

It has been found that relative attributes 1102 are typically morewidely available than categorical attributes 1104 and thus are employedin the following examples. For instance, font families typically orderfonts using relative attributes 1102 including weights and by matchingrelative and italic pairs. On the other hand, categorical attributes1104 such as classification labels (Serif, etc.) are typically ambiguousor not readily available for many fonts 130 although may also be usedwhen available. In the following two relative attributes 1102 areconsidered, font weight and regular-italic pairs although categoricalattributes 1104 are also contemplated as described above withoutdeparting from the spirit and scope thereof.

Font Weight

Fonts are typically organized by families, which are also known astypefaces. A typeface may include fonts of many weights (e.g., frmultra-light to extra-bold or black) and typically have four to sixweights although some typefaces have as many as a dozen. There are avariety of names used to describe the weight of a font in its name whichdiffering among type foundries and designers, but the relative order ofthe weights is usually fixed. For example, relative attributes 1102 maydescribe weights such as: Hairline, Thin, Ultra-light, Extra-light,Light, Book, Normal/regular/plain, Medium, Demi-bold/semi-bold, Bold,Extra-bold/extra, Heavy, Black, Extra-black, and Ultra-black/ultra. Theterms normal, regular and plain, and sometimes also as book aretypically used for the standard weight font of a typeface. Where bothappear and differ, book is often lighter than regular, but in sometypefaces book is bolder.

A designer of a typeface also typically assigns a numeric weightproperty for each font in the family. For instance, the TrueType fontformat employs a scale from 100 through 900, which is also used in CSSand OpenType, where 400 is regular (roman or plain). It is to be notedthat the base weight often significantly differs between typefaces,which means one normal font may appear bolder than some other normalfont even though both are assigned the same 400 weight. For example,fonts intended to be used in posters are often quite bold by defaultwhile fonts for long runs of text are rather light. Therefore, weightdesignations in font names may differ in regard to the actual absolutestroke weight or density of glyphs in the font.

The relative attribute 1102 font weight is usable by the font attributesystem 126 to improve font recognition and similarity by providinganother source of information used to describe characteristics of thefonts. A beginning insight used to improve accuracy of this technique isto first use the weight property to compare fonts from the same family.Since the font feature representation that is learned is shared amongdifferent families, the weight function that is learned may then beapplied to fonts from different families.

Accordingly, the font attribute system 126 employs a metadata attributeextraction module 1106 to extract attributes from font metadata (block1202), such as to extract weights from the relative attributes 1102 ofthe metadata 816 of the fonts 130. Training of the mode using machinelearning is controlled based at least in part on the extractedattributes (block 1204) by a machine learning module 1108. In theillustrated example, the machine learning module 1108 employs a neuralnetwork 110 have at least two machine learning subnets 1112, 1114configured as a Siamese network to learn the weight function and comparefonts. The two machine learning subnets 1112, 1114 of the Siamesenetwork are identical in this example. The end of each machines learningsubnet 1112, 1114 includes a weight prediction layer that is used topredict a scalar values. An additional layer illustrated as theclassifier 1116 positioned “on top” of the two identical machinelearning subnets 1112, 1114 takes the two scalars and forms a binaryclassifier.

To train the machine learning subnets 1112, 1114 of the Siameseconfigured neural network 1110, ordered pairs of images are used thatinclude text rendered using fonts 130 from the same family. Positivesamples are formed such that the first image is of a font with a smallerweight and the other is of a font with a larger weight. Negative samplesare formed such that the first image is of a font with a larger weightand the other is of a font with a smaller weight. Real or syntheticimages may be used as described in the previous sections.

Training sample are organized as a tuple of “(x₀, x₁, y)” for the twoimages “x₀,” “x₁” and binary “(+1/−1)” label “y” indicting positive ornegative pair. The Siamese network generates the scalar weightpredictions “f₀” and “f₁” for two images, and the training objectivefunction is expressed as minimizing a hinge loss as follows:min max(y(w ₀ f ₀ −w ₁ f ₁)+α,0)+γ|w| ₂where “α” is the margin parameter between two weights. A value of “α”may be modulated according to the weight attribute value or set as aconstant. When “f₀” and “f₁” are predicted with a linear neuron in thelast layer of the neural network, the binary classifier 1116coefficients “w₀” and “w₁” are combinable with the weights of the lastlayer. In this way, the linear coefficients together with theregularization term “γ|w|₂” in the above equation may be omitted andtherefore the final objective function 1120 of the font attribute model1118 is expressed as:min max(y(f ₀ −f ₁)+α,0)The scales of “f₀” and “f₁” are normalized after the last linear layerof the neural network 1110 to avoid degenerated solutions or explodinggradients.

An output of the neural network 1110 may be utilized in a variety ofways as part of font recognition and similarity determinations. In oneexample, the font weight attribute is used in training which isadditional information to guide the learning process to form the fontattribute model 1118 that is usable as part of font recognition andsimilarity determinations. In another example, each subnet supplies aweight prediction function that is used to predict a weight (e.g., linethickness) of a font 130. This weight prediction function is consistentacross font families, and may be used to compare fonts from differentfont families once learned.

Regular-Italic Pairs

Fonts 130 typically employ the notion of matching regular and italicpairs. A pair of regular and italic fonts belong to the same family offonts 130 and share a common design, e.g., the italic version slantsslightly to the right. This relative attribute 1102 is also usable toimprove determinations of font recognition and similarity. For example,a Siamese configuration of the neural network 1110 through use ofmatching machine learning subnets 1112, 1114 is used in this instance toclassify matching pairs of regular and italic fonts. Different from theweight property above, a scalar prediction function is not formedbecause it is not used in this example.

As part of training by the machine learning module 1106, training imagesare used that are arranged in pairs. Positive samples are two imagesfrom the same family such that the first image is of a regular font andthe second image is of a matching italic font. Negative samples are twoimages again from the same family such that either both images are ofregular fonts (possibly the same) or both image are of italic font(possibly the same). Training samples may be organized as tuples of“(x₀, x₁, y)” for the two images “x₀,” “x₁” and binary “(+1/−1)” label“y” indicting a positive or negative pair. The Siamese network generatesthe feature vectors “g₀” and “g₁” for two images, and the training ofthe objective function 1120 of the font attribute model 1118 may beexpressed as minimizing a hinge loss as follows:min max(y(h(g ₀ ,g ₁ ;w))+α,0)+γ|w| ₂where “h(g₀, g₁; w)” is a generic classifier with input vectors “g₀” and“g₁” and parameter “w.” The classifier 1116 can be implemented as singleor multi-layer perception neural network. A classifier parameter may beoptimized jointly with the Siamese neural network 1110 usingback-propagation.

Unified Network for Font Recognition and Similarity

In the following, relative attributes 1102 such as the font weight forregular/italics pair are incorporated into a unified training frameworkwith font classification. This unified training framework can apply toboth font recognition and font similarity. Note that this unifiedtraining framework is not limited to these two attributes as thesetechniques are equally applicable to other relative attributes 1102 andeven categorical attributes 1104.

In this example, Siamese configurations of different relative attribute1102 machine learning subnets 1112, 1114 (e.g., for weight andregular/italics pairs) are combined into one and augmented with fontclassification through a multi-task network. To combine the twoattribute Siamese networks into one from the previous sections, forinstance, a single Siamese network is used which has the two tasks ofthe different relative attributes 1102 applied to both subnets. Toaugment with font classification, a softmax classification layer may beadded at the end of each subnet.

Batches of images pairs are used to train the unified multi-task Siameseneural network 1110 in this example. Each batch contains a mixture offour kinds of image pairs. The first kind is two images of differentweights from the same family. The second kind is negative examples ofthe first one. The third kind is two images of matching regular-italicpairs from the same family. Finally, the fourth kind is the negativeexamples of the third one.

For this multi-task network, training samples are formed as a tuple of“(x₀, x₁, y₀, y₁, z)” for the two images “x₀,” “x₁” with font classlabels “y₀,” “y₁” and attribute label “zϵ{0, 1, 2, 3}” indicating oneout of four possible image pair kinds. The Siamese neural network 1110generates the feature vectors “f₀” and “f₁” for two images, and thetraining objective function is expressed as a mixture of hybrid lossesas follows:min softmax(x ₀ ,y ₀)+softmax(x ₁ ,y ₁)+1_(z=0)·max(f ₀ −f₁+α₁,0)+1_(z=1)·max(−f ₀ +f ₁+α₁,0)+1_(z=2)·max(h(g ₀ ,g₁)+α₂,0)+1_(z=3)·max(−h(g ₀ ,g ₁)+α₂,0)

For simplicity, parameter regularization terms and the linearcoefficients used to combine different losses are omitted. Each kind ofimage pair is uniformly sampled during training. If there are fontfamilies with only a single font, those font images may be combined toform a fifth type of image pairs with “z=4.” In this way, the Siameseneural network 1110 may still be optimized with the softmax loss.

Network Structure

Like above, the machine learning module 1108 uses a convolutional neuralnetwork 1110 for training. To train this neural network 1110, theinitial learning rate is set to 0.01, and is reduced to 0.001 in themiddle of training. Batch size is set to 128. Momentum is set to 0.9 andweight decay is set to 0.0005 for each of the layers. Dropout ratioafter “fc7” and “fc8” is set to 0.5. The network structure is listed inthe following table. The multi-task neural network 1110 structure may bereduced to the one of single attribute network by removing some of theloss layers.

Kernel size/ Name type stride Output size tp0 input 111 × 111 × 1 tp1input 111 × 111 × 1 tp0_conv1/tp1_conv1 convolution 11 × 11/2 51 × 51 ×64 tp0_pool1/tp1_pool1 pooling 3 × 3/2 25 × 25 × 64 tp0_conv2/tp1_conv2convolution 5 × 5/1 21 × 21 × 128 tp0_pool2/tp1_pool2 pooling 3 × 3/2 10× 10 × 128 tp0_conv3/tp1_conv3 convolution 3 × 3/1 10 × 10 × 256tp0_conv4/tp1_conv4 convolution 3 × 3/1 10 × 10 × 256tp0_conv5/tp1_conv5 convolution 3 × 3/1 10 × 10 × 256tp0_pool5/tp1_pool5 pooling 5 × 5/5 2 × 2 × 256 tp0_fc7/tp1_fc7 fullyconnected 1 × 1 × 1024 tp0_fc8/tp1_fc8 fully connected 1 × 1 × 512tp0_fcw/tp1_fcw fully connected, 1 × 1 × 1 only used for weightprediction tp0_fcr/tp1_fcr fully connected 1 × 1 × 256 only used forregular-italic prediction fcw Fully 1 × 1 connected, for weightcomparison fcr Softmax for 1 × 2 regular-italic cls0/cls1 Softmax for 1× 4496 font classification

The font attribute model 1118 is obtained by the font similarity andrecognition module 1122, the model trained using machine learning basedat least in part on training data that includes one or more attributesextracted from the font metadata 816 (block 1206). The font attributemodel 1118 is used by the font similarity and recognition module 1122 torecognize the font 1124 used for the rendered text in the image ordetermine similarity of the font used for the rendered text in the imagewith respect to one or more of a plurality of fonts 1126 (block 1208).For example, the font attributes may be used to guide the learning ofthe font attribute model 1118 to improve accuracy and efficiency of fontrecognition and similarity determination techniques. Output of a resultis controlled, the result indicating the recognized font 1124 or thedetermined similarity 1126 in a user interface (block 1210). The userinterface, for instance, may output a result indicating which fonts 130are similar, may be used to apply the font 130 or similar font to rendertext based on the determination, flag text, and so forth.

Thus, in this example the font attribute system 126 uses relative fontattributes 1102 in font recognition and similarity, and may also usecategorical attributes 1104 when available. A unified multi-task Siameseneural network structure is also described that is able to incorporate aplurality of relative attributes, e.g., font weight and regular-italicpair attributes. This unified network applies to both font recognitionand similarity and can be applied to other font attributes.

In one example, a weight prediction function is learned to compare fontsin the same family. After learning, the function can also be used tocompare fonts from different families due to a shared featurerepresentation across font families. Weight prediction and comparison isuseful in font selection. For instance, weight selection may be used tomaps fonts 130 to a single weight axis, find fonts that have larger orsmaller weights than a particular font, and so forth.

Weight prediction may also be used as an aide in font recognition andsimilarity. As illustrated by an example implementation 1300 of FIG. 13,the features learned from a weight attribute network 1304 functionbetter than a font recognition feature 1302 at retrieving fonts withsimilar weight as the input query font.

Additionally, as described in another example above a classifier islearned that is configured to predicts matching regular and italicpairs, and may do so without using metadata. Moreover, theregular-italic classifier may also be used to improve font recognitionand similarity. As illustrated by an example implementation 1400 of FIG.14, the italic attribute features 1404 learned from regular-italicattribute network function better than font recognition features 1402 atretrieving fonts with similar italic style as the input query font. Avariety of other examples are also contemplated without departing fromthe spirit and scope thereof.

Example System and Device

FIG. 15 illustrates an example system generally at 1500 that includes anexample computing device 1502 that is representative of one or morecomputing systems and/or devices that may implement the varioustechniques described herein. This is illustrated through inclusion ofthe image editing module 112. The computing device 1502 may be, forexample, a server of a service provider, a device associated with aclient (e.g., a client device), an on-chip system, and/or any othersuitable computing device or computing system.

The example computing device 1502 as illustrated includes a processingsystem 1504, one or more computer-readable media 1506, and one or moreI/O interface 1508 that are communicatively coupled, one to another.Although not shown, the computing device 1502 may further include asystem bus or other data and command transfer system that couples thevarious components, one to another. A system bus can include any one orcombination of different bus structures, such as a memory bus or memorycontroller, a peripheral bus, a universal serial bus, and/or a processoror local bus that utilizes any of a variety of bus architectures. Avariety of other examples are also contemplated, such as control anddata lines.

The processing system 1504 is representative of functionality to performone or more operations using hardware. Accordingly, the processingsystem 1504 is illustrated as including hardware element 1510 that maybe configured as processors, functional blocks, and so forth. This mayinclude implementation in hardware as an application specific integratedcircuit or other logic device formed using one or more semiconductors.The hardware elements 1510 are not limited by the materials from whichthey are formed or the processing mechanisms employed therein. Forexample, processors may be comprised of semiconductor(s) and/ortransistors (e.g., electronic integrated circuits (ICs)). In such acontext, processor-executable instructions may beelectronically-executable instructions.

The computer-readable storage media 1506 is illustrated as includingmemory/storage 1512. The memory/storage 1512 represents memory/storagecapacity associated with one or more computer-readable media. Thememory/storage component 1512 may include volatile media (such as randomaccess memory (RAM)) and/or nonvolatile media (such as read only memory(ROM), Flash memory, optical disks, magnetic disks, and so forth). Thememory/storage component 1512 may include fixed media (e.g., RAM, ROM, afixed hard drive, and so on) as well as removable media (e.g., Flashmemory, a removable hard drive, an optical disc, and so forth). Thecomputer-readable media 1506 may be configured in a variety of otherways as further described below.

Input/output interface(s) 1508 are representative of functionality toallow a user to enter commands and information to computing device 1502,and also allow information to be presented to the user and/or othercomponents or devices using various input/output devices. Examples ofinput devices include a keyboard, a cursor control device (e.g., amouse), a microphone, a scanner, touch functionality (e.g., capacitiveor other sensors that are configured to detect physical touch), a camera(e.g., which may employ visible or non-visible wavelengths such asinfrared frequencies to recognize movement as gestures that do notinvolve touch), and so forth. Examples of output devices include adisplay device (e.g., a monitor or projector), speakers, a printer, anetwork card, tactile-response device, and so forth. Thus, the computingdevice 1502 may be configured in a variety of ways as further describedbelow to support user interaction.

Various techniques may be described herein in the general context ofsoftware, hardware elements, or program modules. Generally, such modulesinclude routines, programs, objects, elements, components, datastructures, and so forth that perform particular tasks or implementparticular abstract data types. The terms “module,” “functionality,” and“component” as used herein generally represent software, firmware,hardware, or a combination thereof. The features of the techniquesdescribed herein are platform-independent, meaning that the techniquesmay be implemented on a variety of commercial computing platforms havinga variety of processors.

An implementation of the described modules and techniques may be storedon or transmitted across some form of computer-readable media. Thecomputer-readable media may include a variety of media that may beaccessed by the computing device 1502. By way of example, and notlimitation, computer-readable media may include “computer-readablestorage media” and “computer-readable signal media.”

“Computer-readable storage media” may refer to media and/or devices thatenable persistent and/or non-transitory storage of information incontrast to mere signal transmission, carrier waves, or signals per se.Thus, computer-readable storage media refers to non-signal bearingmedia. The computer-readable storage media includes hardware such asvolatile and non-volatile, removable and non-removable media and/orstorage devices implemented in a method or technology suitable forstorage of information such as computer readable instructions, datastructures, program modules, logic elements/circuits, or other data.Examples of computer-readable storage media may include, but are notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, harddisks, magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or other storage device, tangible media, orarticle of manufacture suitable to store the desired information andwhich may be accessed by a computer.

“Computer-readable signal media” may refer to a signal-bearing mediumthat is configured to transmit instructions to the hardware of thecomputing device 1502, such as via a network. Signal media typically mayembody computer readable instructions, data structures, program modules,or other data in a modulated data signal, such as carrier waves, datasignals, or other transport mechanism. Signal media also include anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media include wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared, and other wireless media.

As previously described, hardware elements 1510 and computer-readablemedia 1506 are representative of modules, programmable device logicand/or fixed device logic implemented in a hardware form that may beemployed in some embodiments to implement at least some aspects of thetechniques described herein, such as to perform one or moreinstructions. Hardware may include components of an integrated circuitor on-chip system, an application-specific integrated circuit (ASIC), afield-programmable gate array (FPGA), a complex programmable logicdevice (CPLD), and other implementations in silicon or other hardware.In this context, hardware may operate as a processing device thatperforms program tasks defined by instructions and/or logic embodied bythe hardware as well as a hardware utilized to store instructions forexecution, e.g., the computer-readable storage media describedpreviously.

Combinations of the foregoing may also be employed to implement varioustechniques described herein. Accordingly, software, hardware, orexecutable modules may be implemented as one or more instructions and/orlogic embodied on some form of computer-readable storage media and/or byone or more hardware elements 1510. The computing device 1502 may beconfigured to implement particular instructions and/or functionscorresponding to the software and/or hardware modules. Accordingly,implementation of a module that is executable by the computing device1502 as software may be achieved at least partially in hardware, e.g.,through use of computer-readable storage media and/or hardware elements1510 of the processing system 1504. The instructions and/or functionsmay be executable/operable by one or more articles of manufacture (forexample, one or more computing devices 1502 and/or processing systems1504) to implement techniques, modules, and examples described herein.

The techniques described herein may be supported by variousconfigurations of the computing device 1502 and are not limited to thespecific examples of the techniques described herein. This functionalitymay also be implemented all or in part through use of a distributedsystem, such as over a “cloud” 1514 via a platform 1516 as describedbelow.

The cloud 1514 includes and/or is representative of a platform 1516 forresources 1518. The platform 1516 abstracts underlying functionality ofhardware (e.g., servers) and software resources of the cloud 1514. Theresources 1518 may include applications and/or data that can be utilizedwhile computer processing is executed on servers that are remote fromthe computing device 1502. Resources 1518 can also include servicesprovided over the Internet and/or through a subscriber network, such asa cellular or Wi-Fi network.

The platform 1516 may abstract resources and functions to connect thecomputing device 1502 with other computing devices. The platform 1516may also serve to abstract scaling of resources to provide acorresponding level of scale to encountered demand for the resources1518 that are implemented via the platform 1516. Accordingly, in aninterconnected device embodiment, implementation of functionalitydescribed herein may be distributed throughout the system 1500. Forexample, the functionality may be implemented in part on the computingdevice 1502 as well as via the platform 1516 that abstracts thefunctionality of the cloud 1514.

CONCLUSION

Although the invention has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not necessarilylimited to the specific features or acts described. Rather, the specificfeatures and acts are disclosed as example forms of implementing theclaimed invention.

What is claimed is:
 1. In a digital medium environment to improve imagefont recognition through use of text localization, a method implementedby one or more computing devices comprising: obtaining a model, by theone or more computing devices, that is trained using machine learninghaving a loss function that includes a permissible margin of adifference between positive and negative images as applied to aplurality of training images including: an anchor image having textrendered using a corresponding font; the positive image having text thatis different than the text of the anchor image or text having one ormore applied perturbations; and the negative image having text that isnot in the font type predicting a bounding box, automatically andwithout user intervention by the one or more computing devices, for textin an image received using the obtained model; and generating anindication of the predicted bounding box by the one or more computingdevices, the indication usable to specify a region of the image thatincludes the text having a font to be recognized.
 2. The method asdescribed in claim 1, further comprising recognizing the font of thetext in the received image by the one or more computing devices usingthe generated indication of the predicted bounding box.
 3. The method asdescribed in claim 1, wherein the predicting includes forming aplurality of cropped portions of the image and processing each of theplurality of cropped portions of the image by a trained convolutionalnetwork of the model independently, one to another.
 4. The method asdescribed in claim 3, wherein the predicting further comprisinggenerating the predicted bounding box based on a result of theprocessing of each of the plurality of cropped portions of the image. 5.The method as described in claim 4, wherein the generating is performedby taken an average, a median, or through use of a line fittingalgorithm.
 6. The method as described in claim 1, wherein the predictingincludes resizing the image by the one or more computing devices tocorrespond to an image size of the model.
 7. The method as described inclaim 6, wherein the image size of the model is based at least in parton a size of the plurality of training images used to train the model.8. The method as described in claim 1, further comprising training themodel by the one or more computing devices using the machine learning.9. The method as described in claim 8, wherein the training is performedfor at least one of the plurality of iterations using the plurality oftraining images having text rendered using the corresponding font andperformed for one or more subsequent ones of the plurality of iterationsin which one or more perturbations are introduced to the trainingimages.
 10. The method as described in claim 9, wherein theperturbations includes at least one of noise, rotation, scale, downsampling, shading, rotation, kerning, or cropping.
 11. The method asdescribed in claim 8, wherein the machine learning is performed by theone or more computing devices using a convolutional neural network. 12.The method as described in claim 11, wherein the convolutional neuralnetwork is used as an architecture of the machine learning by the one ormore computing devices and stochastic gradient decent is used as atraining algorithm of the machine learning by the one or more computingdevices.
 13. The method as described in claim 8, wherein the trainingthe model by the one or more computing devices using the machinelearning begins with a random initialization of weights or with apartially trained model.
 14. The method as described in claim 1, whereinthe font to be recognized in the image is arbitrary such that the modelis trainable without using the font.
 15. In a digital medium environmentto improve image font recognition through use of text localization, asystem comprising one or more computing devices including a processingsystem and memory having instructions stored thereon that are executableby the processing system to perform operations comprising: obtaining amodel that is trained using machine learning having a loss function thatincludes a permissible margin of a difference between positive andnegative images as applied to a plurality of training images including:an anchor image having text rendered using a corresponding font type;the positive image having text that is different than the text of theanchor image or text having one or more applied perturbations; and thenegative image having text that is not in the font type; predicting abounding box, automatically and without user intervention, for text inan image received using the obtained model; and generating an indicationof the predicted bounding box that is usable to specify a region of theimage that includes the text having a font to be recognized.
 16. Thesystem as described in claim 15, wherein the font to be recognized inthe image is arbitrary such that the model is trainable without usingthe font.
 17. In a digital medium environment to improve image fontrecognition through use of text localization, a system comprising: meansfor obtaining a model that is trained using machine learning having aloss function that includes a permissible margin of a difference betweenpositive and negative images as applied to a plurality of trainingimages including: an anchor image having text rendered using acorresponding font; the positive image having text that is differentthan the text of the anchor image or text having one or more appliedperturbations; and the negative image having text that is not in thefont type means for predicting a bounding box, automatically and withoutuser intervention, for text in an image received using the obtainedmodel; and means for generating an indication of the predicted boundingbox by the one or more computing devices, the indication usable tospecify a region of the image that includes the text having a font to berecognized.
 18. The system as described in claim 17, further comprisingmeans for recognizing the font of the text in the received image by theone or more computing devices using the generated indication of thepredicted bounding box.
 19. The system as described in claim 17, whereinthe predicting means includes means for forming a plurality of croppedportions of the image and processing each of the plurality of croppedportions of the image by a trained convolutional network of the modelindependently, one to another.
 20. The system as described in claim 19,wherein the predicting means further comprising means for generating thepredicted bounding box based on a result of the processing of each ofthe plurality of cropped portions of the image.